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Abstract — Recently, Vontobel showed the relationship between 
Bethe free energy and annealed free energy for protograph 
factor graph ensembles. In this paper, annealed free energy 
of any random regular, irregular and Poisson factor graph 
ensembles are connected to Bethe free energy. The annealed 
free energy is expressed as the solution of maximization problem 
whose stationary condition equations coincide with equations of 
belief propagation since the contribution to partition function of 
particular type of variable and factor nodes has similar form 
of minus Bethe free energy. It gives simple derivation of replica 
symmetric solution. As consequence, it is shown that on replica 
symmetric ansatz, replica symmetric solution and annealed free 
energy are equal for regular ensemble. 

I. Introduction 

In the context of statistical physics, free energy of disor- 
dered system is central interest. In information theory, the 
a posteriori distribution of low-density parity-check (LDPC) 
codes can be regarded as Boltzmann-Gibbs distributions on 
sparse factor graphs whose free energy is related to the 
conditional entropy of codewords under a received vector JT]. 
In computer science, constraint satisfaction problems (CSPs) 
which can be expressed by sparse factor graphs are important 
theoretical objects. Relation between phase transition phe- 
nomenon and free energy of randomized CSPs has also been 
considered well [2], 0. 

In this paper, we deal with calculation of annealed free 
energy of random sparse factor graph ensemble on finite 
alphabet. Although in many cases quenched free energy gives 
meaningful result e.g., conditional entropy of LDPC codes (T), 
phase transition point of random CSPs [2|, the calculation 
of quenched free energy is often difficult without replica 
method which is mathematically nonrigorous but powerful tool 
of statistical physics. Annealed free energy is also important 
quantity since it can be used for bound of quenched free energy 
and is required in the replica method. 

For many cases |@), in the calculation of annealed and 
quenched free energy, fixed point equations of belief propaga- 
tion (BP) and its density evolution (DE) appear, respectively. 
However, the relationship between BP (DE) and annealed 
(quenched) free energy has not been well understood. Re- 
cently, Vontobel show the relationship between Bethe free en- 
ergy of protograph ensemble and its annealed free energy |]5] . 
From this result, we can connect BP and annealed free energy 



since BP equation is equivalent to stationary condition of 
Bethe free energy (j6). 

The main result of this paper is derivation of annealed free 
energy of any random regular, irregular and Poisson factor 
graph ensembles by using BP equations. The derivation of 
annealed free energy gives the simple derivation of replica 
symmetric solution. It is shown that if the replica symmetric 
assumption is correct, annealed and quenched free energy are 
equal for any regular ensembles. 

II. Factor graph, Gibbs free energy and Bethe 

APPROXIMATION 

In this paper, we deal with factor graph which is bipartite 
graph representing probability distribution [0, [0. Let us 
consider bipartite graph consists of N variable nodes and M 
factor nodes. Let X be alphabet which is common domain 
of variables. For each factor node a, there is a function 
f a : X r " — > M>o where r a denotes the degree of a. The factor 
graph represents the following distribution p on X N . 

1 



where 



Z-=YJ[fa(Xda) 



is constant for normalization, i.e., Y*xP( x ) = 1- Here, xg a 
denotes value of variable nodes connecting a factor node a. 
In the context of statistical mechanics, Z is called partition 
function and — logZ is called Helmholtz free energy. 

When N is large, calculation of Z requires large computa- 
tional complexity. Hence, the approximation of p by simple 
distribution q is often introduced. The following method of 
approximation is written in [6|. For the criteria of approxima- 
tion, Kullback-Leibler divergence is used. 



D (l\\p) —L?W lo g 



«?(*) 
p(x) 



= log Z - £ £ q (x) log f a (x da ) + £ q (x) log q (x) 
x a x 

=: \ogZ + U(q)-H(q) =: \ogZ + T Gibbs (q) 

The quantity U{q), TL(q) and J-Qibbsil) are called internal 
energy, entropy and Gibbs free energy, respectively. 



The approximation using q(x) which is factorized as 
Ili= 1 Qi( x i)> i- e -' x i aie independent, is called mean field 
approximation. The approximation using q(x) which is rep- 
resented as 

Uab a (x da ) 



q{x) 



is called Bethe approximation where i and a represent indices 
of variable nodes and factor nodes, respectively, and where Z, 
denotes degree of variable node i. For Bethe approximation, 
Bethe average energy and Bethe entropy are defined as 

Z4ethe(k«) := -££*ofo fl )log/ a (X 3a ) 

HB S the(bi,b a ) := Y,H h « ix <>" ]h) £ h « {x <>" ) 

+LL(^-i)ft<On)iog*/(*o (d 

i xi 

respectively. Bethe free energy is defined as ^BetheC^n^a) := 
^Bethe(^a) — %Bethe(^' Ai)- ln order to obtain good Bethe ap- 
proximation, minimization of Bethe free energy is considered 
since Bethe free energy is analogy of Gibbs free energy, whose 
minimization is equivalent to minimization of the Kullback- 
Leibler divergence. When we assume constraints, = 1 

for all variable nodes i, Y*x Sa b,j(xg a ) = 1 for all factor nodes 
a, and ^xbafeda) = bi(x) for all factor nodes a and 
variable nodes i e da, the stationary condition of Lagrangian 
is equivalent to condition of fixed point of BP |6). 

III. Annealed free energy of random regular 

FACTOR GRAPH ENSEMBLES 

In this paper, we mainly deal with random regular factor 
graph ensembles. Results for regular ensembles can be gener- 
alized straightforwardly to irregular and Poisson ensembles. 
Let / and r be degrees of variable and factor nodes of 
regular factor graph ensembles, respectively. Let E[-] denote 
the expectation on random connection of edges. Two quantities 
E[logZ] and logE[Z] are called quenched and annealed free 
energy, respectively. The main purpose of this paper is cal- 
culation of limA^oo 1 /ATogE[Z] where N denotes the number 
of variable nodes. The essential idea of calculation is type 
classification of the contribution to partition function 0. Let 
variable-type v denote the type of variable nodes, i.e., there 
exists v(x) variable nodes of value x e X. Let factor-type u 
denote the type of factor nodes, in which the value of factor 
nodes is regarded as the values of variable nodes connects to 
the factor nodes, i.e., there exists u(x) factor nodes connecting 
variable nodes of value x G X r . In this paper, for simplicity, 
factors fa{xda) do not depend on factor node a, and written 
as f(xg a ). Let Z(v,w) be the contribution of assignments with 
variable-type v and factor-type u, and N(v,u) be the number 
of assignments with variable-type v and factor-type u. 



z = £z(v,«)=£^v(v,«) n /(*) 

v.u v,u xex r 



u(x) 



In the sum, the types v and u have to satisfy the consistency 
condition 

= Mz). 

i=l x\x t 

Xj=Z 

The number N(v,u) of assignments with variable-type v and 
factor-type u is 

N \/ L r N MW(*)0' 



Now, we consider the exponent of the contribution of types 
V and fj, where v(x) :=v(x)/N and jj,(x) :=u(x)/((l/r)N), 
respectively. It holds 

lim ~logE[Z(v,/i)] 

Af^co jy 

= ^)-(l-l)«(v) + ; £ MWlog/W 



xex r 



-■ -fBethe(v,ju). 



Hence, 



lim — logE[Z] =max{-F Bet he(v,Al)} 
where, v and ji have to satisfy the following conditions. 
v(x) > 0,Vx e X, n(x)>0,VxeX r 
X>(jc) = 1, £ji(r) = i, 



xex 



xex r 



- r tZ»(x) = v(z),VzeX. 

1=1 X\xj 

Xj=Z 

The last condition is for the consistency between v and jj.. The 
above maximization problem is similar to the minimization 
problem of Bethe free energy. Hence, we can easily understand 
that the stationary condition is similar to the fixed point 
equation of BP. The Lagrangian of the maximization problem 
is 

L(v,H;k,p,T) = -F Be the(V,jU) 



-M I>(*)-i +-p e n(x)-i 



>xex 



\xex r 



-IImW-Mz) ■ (2) 



zex 



= lx\Xj 

Xi=Z 



Lemma 1. The stationary condition of is 

v(x) « m f ^ v (x) 1 



fl{x) °< f(x)Y\m v ^f(xi) 

i=l 



where 



m v ^ f (x) « m/_> v (*) 



(3) 



m f -»(x) -EE /(■*) 11 m v^f{xj). (4) 
i=l x\xi ;=lJ/» 



Here m v - y f(x) and m/_> v (jc) are auxiliary functions satisfying 
L xeX m v ^f(x) = "L xex m f ^ v (x) = 1. 

Proof is in Appendix [A] If f(x) is invariant under permu- 
tation of x, © is simply written as 

r 

m f ^ v (x) oc £ f(x)Y\m v ^ f (x j ). 

X[ =x 

Theorem 2. 

lim -logEfZl 

= max | -logZ/ + logZ v -ZlogZ/ v 

(m v ^ f (x),m f ^ v (x))eS [r 

where S denotes the set of saddle points of the function in 
max, and where 

Z„ := Y,mf^v(x) 1 

X 

r 

z f : =E/( x )ri m ^/( x '') 

X i=l 

Z/v := Y, m f^v(x)m v ^.f(x). 

X 

The conditions of saddle point are (O and @. 

Proof is in Appendix iBl 
Remark 3. Assume that YH=\ /(•*) is constant among all 

Xj=X 

x E X. Then, the uniform distributions m v _>/ (x) and m^_> v (x) 
are a trivial fixed point. Let Nf :— Th e contribution 

Z(v,/x) evaluated at uniform v and /I is 

I N f 1 1 I N f 

-log-^+log-T-r-Zlog- = log<7+-log-e. (5) 
r q q 1 g r q r 

When /(x) E {0, 1 }, i.e., the problem is the CSP, Z is the num- 
ber of solution and Nf is the cardinality of {x E X r \ f(x) — I}. 
In this case, we call the quantity (0 design rate. If the uniform 
V and fi maximize Z(v,jtt), the expected number E[Z] of 
solution is about 



q [ V 

Roughly speaking, this implies that all constraints are inde- 
pendent. This solution is called paramagnetic solution in 0, 
in the context of replica symmetric solution. 

The generalization for irregular and Poisson ensemble is in 
Appendix [H] 

IV. Contribution to partition function of fixed 

VARIABLE TYPE 

We now consider the contribution to partition function of 
regular factor graph ensemble with fixed variable type. More 
precisely, we consider Z(v) := £ m Z(v,h). It holds 

lim ~logE[Z(v)] =max{- J F Be the(v,M)}- (6) 

The function — / 7 Bethe(v,/i) is a concave function with respect 
to ji. Since the equality constraints are linear, the problem 



is essentially a maximization problem of a concave function 
without constraints. Hence, it can be solved numerically by 
the Newton method. 

Lemma 4. The stationary condition of (O is 

r 

p(x) oc f(x)Y[m v -+f(x i ) 

where 



;=1 



v(x) oc h(x)m f ^ v (x) 1 (7) 
m v ^f(x) °c h(x)mf^. v (x) l ~ l (8) 

r r 

m Mv (x) ~ £ £ /(*) EI "V-tf (•*/)• ( 9 ) 

i=lx\xj j=l,j& 

Xj=X 

Here m v ->f(jc), ot/_>. v (jc) and h{x) are auxiliary functions 
satisfying Y.xex m v^f{x) — Y.xex m f^v(x) = 1, and h(x) > 0. 

Proof is in Appendix [D] Since h(x) is arbitrary auxiliary 
function, nif^ v (x) 1 in (0 and mf_ >v (x) i " in ([8]), can be 
replaced by mf^ Y {x) k and wif-^fjc) , respectively for any 
k > 1 . Here, we chose A; = / since we can obtain the following 
simple result. The stationary condition for magnetic field 
model in Appendix [C] and Lemma [4] are the similar although 
while in the problem for magnetic field, h(x) is given and v(x) 
is variable, in this problem, h(x) is variable and v(x) is given. 

Lemma 5. 

Urn llogE[Z(v)] 

(I 

= max < -logZ/ + logZ v — /logZ/ v 

(m f ^ v (x),m v ^ f (x)Mx))eS { r 



£v(x)log/z(x) 



(10) 



max 

(m/->vW. m v->/M)es 



| J logZ, + £v(*) logZ v (x) - ZlogZ/v 



-W(v) 



where S denotes the set of saddle points of the function in 
max, and where 



x i=i 

Zv(x) :=m f ^ v (x) 1 
Zy :=£/i(x)Z v (x) 

Z/ v := £m/-> v (*)»»v--»/(*)- 

jr 

77ze conditions of saddle point are Q, ([8]) ant/ 

The annealed free energy of magnetic field model in Ap- 
pendix |C] is obtained by the Legendre transform of the above 
result. It can be easily verified from (TT0V 

While the maximization problem (O can be solved by the 
Newton method, Lemma [4] gives the efficient algorithm. First, 



{m^ v (x)} xe x are initialized. Then, messages are updated by 



v(x) 



.(' 



i=lx\xi 3=\,&i 

Xj=X 

iteratively. After sufficient iterations, messages are substituted 
to 

I filogZ / +£v(je)lDgm/_ fv (*)-logZ/v^ +H(v). 

Note that the degree I of variable nodes does not appear 
in the iterations and only appear as the factor of the first 
term in the last equation. This algorithm does not necessarily 
converges. The example of problem for which the above BP- 
like algorithm does not converge is shown in Section [VT] 

V. Moment of partition function and replica 

METHOD 

In this section, we deal with moments of partition function 
which is useful for some purposes. One of the most successful 
result of use of moment is the second moment method i.e., 
for nonnegative random variable Z, P(Z > 0) > E[Z] 2 /E[Z 2 ]. 
Using this method, lower bound of SAT-UNSAT threshold is 
obtained [7 |. The other use of moment is the replica method 
which is not rigorous but powerful tool of statistical physics 
for calculation of quenched free energy. The basic idea of the 
replica method is representation of E[logZ] as the derivative 
(<?logE[Z"])/<?«|„ =0 . It holds 

lim lE[logZ]=limilim^ffl. 
If the exchange of the limits is admissible, 



lim -EflogZl = lim - lim -logE[Z"l. 



In the replica method, 



lim -logElZ'l 



(ID 



(12) 



have to be evaluated. Usually, (fT2l is evaluated only for n G N 
such that dependence on n is analytic. Then, the right-hand 
side of dTTb is evaluated by ignoring that n should be natural 
number (3). 

Since Z" can be regarded as partition function of factor 
graph on alphabet X" and factor FI^Li /(■*)> me exponent of 
moment is also calculated in the same way. Here, E X r 
denotes vector (jcj'\ . . . ,xf > ) where xj is j-th elements of x 6 
(X") r and x^ denotes z'-th element of xj £ X" . 

Corollary 6. 



lim -logEfZ"! 
= max { -logZf + logZv — /logZf v ^ (13) 

(m Mv (x),m v ^ f (x))eS [r ■' 1 



where S denotes the set of saddle points of the function in 
max, and where 

Z v := Y, m f^v(x) 1 
xex n 

z f-= e (n/(* w ))rim^/(*<) 

xe(x"Y v=i / i=i 

Z/v := 52 mf^ v (x)m v -^.f(x). 
xex" 

The essentially same result for LDPC codes was obtained 
in JS] (Eq. (5.2)). In (8), it is explained that the replica sym- 
metric assumption says that distributions m v ^f{x^\ . . . 
and mf^ v (x^\ . . . ,x^ n ') which are invariant under permutation 
dominates E[Z"]. Furthermore, the representations 

f " 

m v ^ f {x) = / Y\M v ^ f {xj)d<S>{M v ^ f ) 



l f- 



v (x) = J f[Mf^ v (xi)d&(Mf^ v ) 



are assumed where <I> and <t> denote probability measures on 
V(X), i.e., 4> and 4> are elements of V{V{X)). Here, V{A) 
denotes the set of probability measures on a set A. 

Lemma 7. 

-F RS = max {-(logZf) + (logZ v )-Z(log2/ v )l 
(*,*)gs I r J 

where S denotes the set of saddle points of the function in 
max, where 

z * E Il<vW 

xEXi=l 

Zf-= E mt\ M v\M) 

xex>~ i=l 



Z fv := £ M v ^ f {x)M Mv (x) 



xex 



where {M^y}, = i ... r and {Mj^Jy r . j are i.i.d. random 
measures obeying <t> and <I>, respectively, and where (■} denotes 
the expectation with respect to the random measures. The 
saddle point conditions are 



f(0 



Lxexri,Z{M { ;l v (x) 
I.xeX'x D =x f(x)Wj=i j^p M { J\ f (xj ) 
LcexrmUUj^M^fixj) 



<4> 



where D denotes the uniform random variable on {1,2, ... ,r} 
which is independent of any random variable, and where M ~ 
<t> denotes that a random measure M has a law O. 

Proof is in Appendix|E] This derivation of replica symmetric 
solution is simpler than previously known ones (9), JU, iflOl 
in which complicated tools are used e.g., integral expression 
of delta function. Another advantage of this paper is that we 



can understand why the saddle point equation in the replica 
symmetric solution is equal to the DE equation. 

When f(x) is invariant under permutation of x, the fixed 
points for annealed free energy in Lemma Q] are also fixed 
point for RS saddle point equation as delta distribution. From 
inclusion relation of domains of max in Theorem |2] and 
Lemma |7] — Frs > limv-^oo l/iVlogE[Z]. On the other hand, 
from Jensen's inequality, E[logZ] < logE[Z]. We now obtain 
the following theorem. 

Theorem 8. Assume f(x) is invariant under permuta- 
tion of x. If replica symmetric assumption is valid i.e., 
-F RS = limjv^ool/MEllogZ], then \xm N ^> l/AHE[logZ] = 
limjv^ l/iVlogE[Z]. 

This result is well known for regular LDPC codes ifTUI . 
When we believe the replica method, even if replica 
symmetric assumption is not valid, intuitively — Frs < 
lim;v^oo l/ATEflogZ] holds, since the replica symmetric as- 
sumption restrict the domain of maximization problem. How- 
ever, generally — Frs > limA^oo 1 /ME [log Z] can be hold fTTl . 
Hence, Theorem [8] requires the replica symmetric assumption. 

This result can be generalized for random factor model 
straightforwardly. For the random magnetic field model in 
Appendix ICl lim^^oo 1 /A/logE{/,.} [E[Z"]] have to be evaluated 
for the replica method. This quantity can be calculated easily 
by Theorem|2]by replacing h(x) by E/, In this case, the 

relation — F R $ > lim^^oo 1/AflogEr/,.} [E[Z]] does not hold. 

VI. Applications 

In this section, an example of binary CSP is shown. The 
factor is 

/(?) = { ' u 2 , 2 forxe{0,l} r . (14) 

I 1 , otherwise 

This factor is considered to prevent assignment from including 
half numbers of Os and Is. The number of solution of fixed 
variable type is calculated by the BP-like algorithm shown 
in Section [IV] The calculation results for (10,20) regular 
ensemble are shown in Fig. Q] The horizontal axis shows the 
relative number of Is in solutions. For all k, v(l) = l/2 is 
not peak of growth rate. This means that the paramagnetic 
solution is not solution of the maximization problem in The- 
orem [2] For k = 3, algorithm does not converges in region 
including v(l) = 1/2. When v(l) = 1/2, the paramagnetic 
solution m v ^f(x) = nif^ v (x) = 1/2 for x = 0,1 is a fixed 
point of the iteration. In Appendix [I] the stability condition 
of the paramagnetic solution when v(l) = 1/2 is shown. It 
is confirmed that the stability condition is violated for r = 20 
and k = 3. 

VII. Conclusion 

The annealed free energy of any regular, irregular and 
Poisson factor graph ensembles are shown. The expression of 
annealed free energy includes the BP equation. This result 
gives simple derivation of replica symmetric solution. As 
consequence, on the replica symmetric ansatz, it is shown that 




Fig. 1. Growth rate of the (10,20) ensembles. 



annealed and quenched free energy are equal for any regular 
ensembles satisfying that f(x) is invariant under permutation 
of X. 
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Appendix A The equation in Theorem [2] is obtained from the equality 

Proof of LemmaQ] ^ , w 

_. Z, _ Lxex m f^v{x) _ 
Partial derivatives of the Lagrangian (|2) are ~ — TTTT ~~ f v ~ 

^v-tf Lz.ex m f^v(z) 

— — — = (/ — l)(log v(x) + 1) + A — lx{x) On the other hand, let us consider the function 

dv(x) 

3^- = --(log M W + 1) + - Iog/(x) + -P + -Y z(x t ) F ({^fh {m f ^ v }) ■■= l -l°gZf + logZ v HogZ /v 

djl{x) r r r r r-^ 

for any non-negative functions mf-+ v (x) and m Y ^f(x). This 
We can define m v ^ f {x) and m/-> v (x) which satisfies quantity is invariant under scaling of {m f ^ v (x)} xeX and 
LxeA ,m v^/( x ) = Hxex m f-*v{x) = 1 as {mv->/(x)}^e^. Hence, without loss of generality, we can as- 

1 sume Ya m f-yv(x) — E* m v->/(*) = 1- Since the first derivatives 

T(x)=:logm v ^ / (x)=:log- m/_> v (x) are 

where Z v _>/ is normalization constant. d F ( {m v _>/- } , { ot/-» v } ) _ m/_» v (^) _ m v ^f(x) 

dnif^ rv {x) Z v Z/ v 



i 



z-i J VZv^/, 

H(x) = eXTp{-l+p}f(x)Y[m v ^ f (x) m f ^ v (x) 
i=i — i — = 



From the normalization conditions, X and p are determined 
uniquely. From the consistency condition of v(x) and n(x), it the saddle P oint condition is 
holds . Zf v j_j 



Z /v 



1 r 1 r 1 



m v ^ f (x) = —m f ^ v (x) 



■■=i ; \ 4 z / j=i ^ ^w = -^EE/W n «v-/(*A 

v^/W-EE^/w n ^/fe^Tm/^w 1 . ' . . 

i^ix\x ■ f j=ij^i v Although these points take minimal on any lines parallel 

x ' =x to axis of {my^ v (x)} or {m v _^ (jc)}, these points are not 

Z " l TTf(x) FT m f fx-1-« f M necessarily minimal. 



Z/Z, '^ / r ! '= 1 j = 1 '^ ! ' Appendix C 

Magnetic field model 

Appendix B 

Proof OF Theorem[2] Although we have only considered the random regular factor 

t . • , graph ensembles, the method can be generalized straightfor- 

Let us consider b r b b 

wardly to many ensembles. As a simple example, we introduce 

-H(p) - (I - l)n(v) + - Y, M(*) lo g/(*) (15) the random regular factor graph with magnetic field. 
r r xex r j 

evaluated at v and jj, at the stationary point i.e., they satisfy = ^ J^JX0* : ^a) X^I^^ 1 ')' 

Lemma Q] 

(\ Here, there are degree one factor nodes for each variable node. 

/(*) J^mv^/(jcf) In the same wa Y> me annealed free energy can be calculated. 

' =1 ' 1 

= ^ogZ f -^(x)lo g f(x)-lZHx)lo g m^ f (x) ^N l0gE[Z{{v ^^ 

/, v , „ * . , =-H({At})-(/-l)H({v}) + - £ AiWlog/(x) 

' „ * X ' + £v(*)log/i(*). 

+ /logZ v _ ) ,/ T 

_, , Lemma 9. 

(/ - l)H(v) = (/ - lJlogZy - (/ - 1) £ v (x) logm/^vW 

A lim -logEFZl 

Hence, (B) is N 

Z 



, _ , , _ n Zv = max < -logZf + logZ v — ZlogZf,, 

logZ / + logZ v -/log^- (m^W^WjGsir S ; S S /l 



where S denotes the set of saddle points of the function in _ q _ l)(lpg v(x) + 1) + A + y, TJfcflfcM ~ 

max, and where dv(x) k=l 

Z„ := Y^h{x)m f ^ v {x) 1 = - l -(logn(x) + 1) +^log/(x) + l -p 

Z f := £/(*) f[m^ f {xi) +-E Ckc k (x) + - £ 

jc /=! r k=\ r 1=1 



Z /v -Imz-wW'Bh/W Let 



77,r v/«//V;„«/t /.s T (j) = : logm v ^/(jc) =: log ( -L f JK* W ) m/_» v (*)'-' 



m v ^f(x)°ch(x)mf^ v (x) 1 1 
»/-»(*) « £ E /(*) EI £fc =: loggt 



= 1jc\.y; j=i,#i 

Xj=X 



The rest of the proof is same as the proof of Lemma Q] 



The above stationary condition is related to the stationary 
condition of maximization of the contribution Z(v,jtt) with APPENDIX E 

fixed variable type v in Section [TV] Proof of Lemma|7] 

We use the following relation. 



Appendix D 

Proof of LemmaH] ,. 1, ,.„, ,. 

hm - log (A ) = (log A) 

Generally, when we have additional linear constraints n 

L where A is a random variable and (•) denotes an expectation. 
ai c {x)v(x) — bk, for k = 1,2, . . . ,s 

xex ( r " V 

£ c k (x)fl(x) = d k , for k= 1,2,...,? Zy = £ / n^/(^)dP 

x r E.x xex" y i=i / 

in the maximization problem of -F B ethe({v}, {ju}), the sta- /• r ( ' \ ( ' {j) \" 

tionary condition is = j "' j 1 JJ 7 ) I J 

MW-fn^ w Vwn"^/w Henc6, i 



VJfe=l 

(,<»*(■*) 



lim - logZ,, = (logZ v 

«->.0 n 



v(x) oc | ]~[/!^ (A) J m Mv {x) 1 The derivation of Z/ and Z /v are similar. 

' The derivation of the saddle point equations are omitted 



\k=l 



V*/M - f\hf X) m/^M'- 1 SinCC * " Strai S htforWard - 



\ k=l 



Appendix F 
Regular LDPC codes 



=i*Vi V^ 1 / Corollary 10 ((Litsyn and Shevelev, 2002), (Burshtein and 

Miller, 2004)). Growth rate of (l,r)-regular LDPC code en- 
where {h k > 0}*=!,...^ and {g k > 0}k=i...., t are auxiliary vari- semble is 
ables. 

Proof: G((») = -log 1 + 



L(v,ju;A,p,T7,^,T) = -F Be the 



i-2 
log 



+A £ v(*)-i + £ tjJ £ fljt(*)v(*)-A t 

W / *=i \xex J _/i og i2Zl_ (0 '/i 

"P f E C *W^ W - * | w/tere 0)' := 1 - 2ffl and 

r \xex r / r k=\ \xex r J 

d = tanh (h + 1 tanh~ 1 (/ ) ) 



E^)|^tE/»W-/vw|. v ," ■ 



^' =1 ^4 / z^tanh^ + ^-^tanh-^y)). 



This result can be easily understood from Lemma [4] and [5] 
by observing the following corresponding s, 

o' = v(0)-v(l), h = {-l) x \ogh{x) 

z = m v ^ f (0) - m v _>/ ( 1 ) , y' = m f ^ v (0) - m/_>v ( 1 ) 

and 



subject to 



Zf 

Z,, : 



log 
log 



l+z" 



i+y 



i-/V 



1+yV 



Zfy = log 
V v(x)log/z(x) = co'h. 

x 

This result is also obtained by using the combinatorial method 
in and change of variables (01 



1 



■ log*, 



v 



1-y 



1-Z 



2 ° ' l+y l+z 

But the proof of this paper is much more meaningful. 

Appendix G 
Random magnetic field model 

In this appendix, we consider the random magnetic field 
model. 

p(* I {hi}) = zm^UfMUHxi) 

Z{{hi})=YX\f{xda)X\hi{xi). 

x a i 

Here, {hi} independently and identically distributed according 
to the distribution Pn(h) on a finite set H of nonnegative 
function on X. In statistical physics, hi(xi) represents ran- 
dom magnetic field. As a posteriori probability of LDPC 
codes, hj corresponds to output of a channel. We now 
consider limbec l/jVE w [logl[Z({A/})]]. Since E[Z({/i,-})] 
depends on {hi} only through the type of {hi}, and since 
l/NlogE[Z({/z,-})] = 0(1) for any {hi}, we only have to deal 
with typical {hi}. Let v(x,h) denotes the number of variable 
nodes of value x and whose corresponding factor is h. The 
factor-type u(x,h) is defined in the same way. Then, it holds 

z=i>(v,io n n h{x) v(x,h) 

v.u {x.h)eX r xW (x,h)eXxH 

For typical {hi}, it holds 



NPhQi) 



E[N(v,u)} y{ v (x h }} xex j \{u(x,h)}^) eX r xn 

Il(x,h)eXxu( v ( x , h ) l ) ] 



'-N 



(my. 



Hence, the problem is maximization of 
-H(ji)-(1-1)H(v)-H(Pb) 



+ ■ 



I 



/i(*,A)log/(x)- 



(x.h)eX r xU'~ 



Y v(x,h)\ogh(x) 

(x,h)eXxH 



v(x,h)>0, n(x,h)>0 
J £v(x,h)=P H (h), J>(M) = 1 



xh 



~Y Y M(*>*) = v(z,*),Vz€#,Ae«. 

r 1=1 (x,h)\(x„hi) 
Xj=z,hj=h 



Lemma 11. The stationary conditions are 

r 

H(x,h) <* f(x)Y[ m v^f (xi , hi) 
v(x,h) oc g(h)h(x)m f ^ v (x) 1 

where 

P H {h)~g{h)Y,Kx)m Mv {x) 1 (16) 

xex 

m v ^ f (x,h) oc g(h)h(x)m f ^ v (x)'~ l 

m v ^f(x)ocY j m v ^f(x,h) (17) 

r r 

m f ^ v (x) oc Y Y /(*) I~[ ( 18 ) 

Xj=X 

Here m v ^f(x), ntf^ v (x) and are auxiliary functions 
satisfying ^xeX m v^f{x) = LxeX m f^v( x ) = h and g(h) > 0. 

Lemma 12. 



Urn E {/i . } [logE[Z]] 



max < -logZf + logZ v — ZlogZ/- v 

( ffl ^,W,/n M /W,sCi))e5 r 



X><*>,^} 



, P ax , ^ ^ -logZ/ + y>ff(/z)logZv(/0 



- Hog Z /v j 

where S denotes the set of saddle points of the function in 
max, and where 

r 

Z f '= E f( X )T\ m v^f( X i) 
xeX r i=\ 

Z v (h) := Y h(x)mf^ v (x) 1 
xex 

Z v := Y g(h)Z v (h) 

hen 

z fv ■= Y m f^v{x)m v ^ f (x). 
xex 

The conditions of saddle point are (1161 l to ( 118b . 



Appendix H 
Irregular and Poisson ensembles 

A. Irregular ensemble 

The result can be generalized for irregular ensembles. Let 
T> v and V c denote the set of degrees of variable nodes and 
check nodes, respectively. Let L, and Rj denote the degree 
distribution of variable nodes and check nodes from node 
perspective for i G D v and j G T> c , respectively. Assume the 
factor corresponding to degree j factor nodes is fj(x) for 
j G T> c . Let v(i,x) denotes the number of variable nodes of 
degree i and value x. The factor-type u(j,i,x) is defined in the 
same way. 



E[tf(v,«)]=n (r 

itA \M^ x )}xex 



n 



1/(1) 



Wtf) NR J 



jev c v{ M J,<>}(,» 6 p/ x ^ 
The problem is the maximization of 

Km ilogE[Z(v )M )] 
L'(l) 



(NL'(l))\ 



mTTT E E L,(i-m(Vi)-L'(l)n(Li(i 

L'(l) 

^TTTT E E M(M*)l°g/K*) 



subject to 



v{i,x)>0, n(j,i,x)>0 
£v(j»=L,-, £ n(j,i,x)=Rj 

{ix)ev>,xxi 



xex 



L'(l) j 

■577TT E E E liU,i,x) = iv(i,x). 

We obtain the following stationary conditions. 

j 

v(?» oc l(i)mf-y v (x)' 
U x l{i) E m /->vW 

.7 

m v ^f(i,x) °< il(i)mf^ v (x) 1 ^ 1 

mv->f(x) oc £ il(i)m f ^ v (x)'~ l 
iev v 



(19) 
(20) 

(21) 



m / ^ v ,(x)oc £ £ E r U)fj( x ) il m ^f{ x k) (22) 



Lemma 13. 

lim — logEfZl 



) L'(l) 



max 



logZ/ + logZ v 



(0 



max 

(m v _>/ (*),»!/_> 



(x))es [«(,!) ; - 62 , c i6T , v 



■L'(l)logZ /v 



(23) 



where S denotes the set of saddle points of the function in 
max, and where 



ZfU) 

Z v {i) 

Zv 

z /v 



xexi k=i 

■■ £ m f ^ v (x)' 
xex 

: £ /(/)Z V (/) 

= E Kj)Z/0') 

y'e£>c 

: m f^v{x)m v ^ f {x) 
xex 



The stationary conditions are d!91 l fo (I221 l. 

This second expression ( l23l is equivalent to the equations 

in a. 

B. Poisson ensemble 

In this subsection, we deal with Poisson ensemble. There are 
A/ variable nodes and aN factor nodes. The degree of factor 
node is k. For each factor node, connecting variable nodes are 
chosen independently and uniformly from N(N — 1) • • • (N — 
(k—l)) ways. In the same way as other ensembles, we obtain 



E[/V(v,w 

xn 

xex k 



N \ / aN 
{v(x)}xexj \{u(x)} xeX iy 

nxexv( x Mx)-l)---(v(x)-(Nx(x ) 1)) ^ " W 
/V(/V -!)•••(/¥- (*-l)) 



where N x (x) denotes the number of x in x. The problem is 
maximization of 
1 



lim -logE[Z(v,M)] 

N— A 



= aH(ii) + H(v) + a £ /i(x)log ( [[v^) 

1=1 



jev c t=l xeX jj\ Xt 

x,=x 



-a £ A*(*)log/(x) 

xex k 

-aV(ii\\v k ) + U{v) + a £ M(*)log/(a 

xex k 



subject to 

v(*)>0, mW>o 
£v(*) = i, £mW = i- 

This is also similar to the minimization of Bethe free energy 
since ([D is also written as 

n Bethe (b h b a ) = -EEfc a fo a )i g -My 
-EE & «'(*i) lo gM*«)- 

i JC," 

The derivation of the following lemma is omitted for lack of 
space. 

Lemma 14. 



By linear approximation, 

v(x) f 8{x) 



m f^ v ( x ) 



■&(8(x) 



= l-q8(x) + ®(8(* 



i=l x\xj j=lj^=i 



ie/(*) n (i-^(x ; )+©(5(x,) 2 )) 



lim -logE[Z(v,At)] 



max < alogZ/ + logZ v 

(m/_^ v (x) ,my-> f (x),e)eS I 



- e £ m v _>/ (x)m f ^ v (x) j (24) 



where S denotes the set of saddle points of the function in 
max, and where 

k 

z /:=E/( x )n mv ->/(*'') 

X (=1 

Zv := £exp{em/_> v (x)} 
The conditions of saddle point are 

(X ^ M^M 

m Mv {x) = — £ E f(*) II m ^f( x j) 



'f i=l XGX k j=lJ¥ : > 



m v ^f(x) oc exp{em/^ v (x)} 

, , (emf^Jx)) 2 
= 1 + em f ^ v (x) + + ■ ■ ■ . 

Here, e can be regarded as mean of Poisson distribution 
expressing the degree distribution of variable nodes. Note that 
the third term of (l24l evaluated at saddle points is ak. 

Appendix I 
Stability of the paramagnetic solution 

Assume L| =1 L*^. /(*) is constant among all x€ X, Let 

Xi=X 

v(x) — m.j-_^ v (x) — m%_yf(x) = 1/q for all x e X. Let us start 
the algorithm in Section [IV] from 



m f ^ v (x) oc nif^ v ( x ) + 8(x). 



i=lx\xj 

Xj=X 



xex 



Let 



, LULxvJWfcU^Axj, 

_|_ Xj=X I 

q Li=il.x\x, f(x) 

Xj=X 
X:=X \ 



E 

xex 



8+(x) := 



Li = iLx\xJ(x) 



We now consider the linear operator A defined by 
A({8(x)} xe x) = {8 + (x)} x ex ■ The all 1 vector is a eigenvector 
of A with eigenvalue — (r — 1), The stability condition of the 
paramagnetic solution is that absolute values of eigenvalues 
of A not corresponding to the all 1 vector are smaller than 1 . 
For the binary CSP (fl4l . the matrix A is a symmetric 2x2 
matrix where 



An =A 22 = - 



A12 =A 2 i = 



2L i t fc - i ( r 7 i )+(r-i) 



The eigenvalues of A are An +A12 and An — A 12 whose 
eigenvectors are [1 l] T and [1 — l] T , respectively. We can 
easily confirm that 



-Ai2 = -(r-l) 



-Al2 = 



2Efco fc - 1 r7 1 )+(p 



Hence, the stability condition is 

(gj)g*-l) 



For r = 20 and k= 1,2,3, the left-hand side of the condition 
is 0.23883, 0.859049 and 1.825917, respectively. This result 
is consistent with the numerical calculation result in Fig. Q] 



< 1. 



